1. Field of the Invention
In at least one aspect, the present invention relates to process for improving data flow in a PCI-Express fabric.
2. Background Art
PCI-Express (“PCIe”) has become the most prevalent input/output (“I/O”) interconnect technology for a wide range of computer systems, from workstations up through high-end servers. The technology has many built-in features that provide a high level of system reliability, accessibility, and serviceability (“RAS”).
PCIe utilizes a credit based flow control scheme in which a device advertises the number or amount of space available in its buffers. PCIe flow control is done on a per-hop basis, i.e. the flow control is local between a pair of devices. The PCIe specification defines a separate flow control resource for each of the following types of packets: posted request packets, non-posted request packets, and completion packets. A device keeps separate flow control credit counters for header and data, for each of the three packet types. Accordingly, a PCI device contains 6 different counters. Flow control credits are returned from the receiver to the sender periodically, as the receiver frees buffer space for each respective packet type. The return of credits is done using Update Flow Control (“UpdateFC”) Data Link Layer Packets “(DLLPs”), where there is a separate UpdateFC type for each of the three types above. A given UpdateFC specifies credits for both header and data, in two separate fields.
The PCIe architecture defines an optional flow control timeout error, which fires when a sender has not received an UpdateFC DLLP of a given type for a specified period of time. For the PCIe 2.0 spec., that time is 200 usec. The timer is reset by the receipt of an UpdateFC DLLP. When the timer expires, the error is logged and the Physical Layer is instructed to retrain the link. While this error is designed to catch hardware faults that prevent a device from sending UpdateFC packets, it cannot detect the case where UpdateFC packets are received, but the credit value returned in the UpdateFC packet never changes. In other words, a device may send UpdateFC packets on a regular basis but due to congestion or faults in the fabric downstream of the device, it never frees space in its buffers and so never returns credits to the sending device. In such a situation, forward progress is hindered because the sending device is not able to send packets (of a given type, or perhaps of multiple types).
Although the present implementations of PCIe work quite well, there are several conspicuously missing features in the current generation of the protocol. An example of such a desirable feature is a method for detecting and signaling when a PCIe device is failing to make forward progress. Forward progress in this context means that a device is able to issue transaction requests and have them completed in a timely manner. The same device is also able to issue responses to transactions for which it is the target in a timely manner. Forward progress can be stalled when a device does not have flow control credits needed to issue packets onto the link—whether they are requests or completions (responses to requests issued earlier to the device).
Accordingly, for at least these reasons, there is a need for methods that facilitate data movement in a PCI Express connection.